Papers with vision-language models
Dual-Axis Generative Reward Model Toward Semantic and Turn-taking Robustness in Interactive Spoken Dialogue Models (2026.acl-long)
Copied to clipboard
Yifu Chen, Shengpeng Ji, Zhengqing Liu, Qian Chen, Wen Wang, Ziqing Wang, Yangzhuo Li, Tianle Liang, Zhou Zhao
| Challenge: | Reinforcement learning (RL) has improved text- and vision-language models, but its application in SDMs is hindered. |
| Approach: | They propose a dual-axis Generative Reward Model that provides semantic quality and interaction timing for SDMs. |
| Outcome: | The proposed model achieves state-of-the-art performance on interaction-quality assessment across a wide spectrum of datasets. |
SCoPE VLM: Selective Context Processing for Efficient Document Navigation in Vision-Language Models (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing methods for document understanding are memory-intensive and impractical for local deployments. |
| Approach: | They propose a document navigation expert that leverages a Chain of Scroll mechanism to selectively and recursively navigate documents, focusing exclusively on relevant segments. |
| Outcome: | The proposed method reduces memory usage and effectively models human-like reading behaviors. |
ImageInWords: Unlocking Hyper-Detailed Image Descriptions (2024.emnlp-main)
Copied to clipboard
Roopal Garg, Andrea Burns, Burcu Karagol Ayan, Yonatan Bitton, Ceslee Montgomery, Yasumasa Onoe, Andrew Bunner, Ranjay Krishna, Jason Baldridge, Radu Soricut
| Challenge: | generating accurate hyper-detailed image descriptions is challenging for vision-language models trained on web-scraped image-text. |
| Approach: | They propose a data-centric framework for generating hyper-detailed image descriptions using web-scraped image-text. |
| Outcome: | The proposed framework improves on human evaluations on the data, even with only 9k samples. |
Rad-Flamingo: A Multimodal Prompt driven Radiology Report Generation Framework with Patient-Centric Explanations (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing reports are labor-intensive and expert-intensive, resulting in inconsistencies and a lack of patient-centered insight. |
| Approach: | They propose a multimodal prompt-driven report generation framework that integrates diverse data modalities to produce comprehensive and context-aware radiology reports. |
| Outcome: | The proposed framework improves report quality, improves understandability and could foster better patient-doctor communication. |
Router-Suggest: Dynamic Routing for Multimodal Auto-Completion in Visually-Grounded Dialogs (2026.eacl-industry)
Copied to clipboard
| Challenge: | a task that grounds predictions in multimodal context is essential for chatbots, chatbot systems and healthcare consultations. |
| Approach: | They propose a task that grounds predictions in multimodal context to better capture user intent. |
| Outcome: | The proposed task can be used to predict upcoming characters in live chats using partially typed text and visual cues. |
Developing Japanese CLIP Models Leveraging an Open-weight LLM for Large-scale Dataset Translation (2025.naacl-srw)
Copied to clipboard
| Challenge: | lack of large-scale open Japanese image-text pairs poses a significant barrier to the development of vision-language models. |
| Approach: | They construct large-scale Japanese image-text pairs using machine translation and pre-trained CLIP models on a Japanese dataset. |
| Outcome: | The results show that pre-trained models achieve competitive average scores on Japanese culture tasks compared to models of similar size. |
Expedited Training of Visual Conditioned Language Generation via Redundancy Reduction (2024.acl-long)
Copied to clipboard
| Challenge: | EVLGen is a framework for visual-language pre-training with high computational demands. |
| Approach: | They propose a streamlined framework for the pre-training of visually conditioned language generation models with high computational demands. |
| Outcome: | The proposed framework accelerates training of vision-language models by a factor of 5 without compromising performance. |
MiSCHiEF: A Benchmark in Minimal-Pairs of Safety and Culture for Holistic Evaluation of Fine-Grained Image-Caption Alignment (2026.eacl-short)
Copied to clipboard
Sagarika Banerjee, Tangatar Madi, Advait Swaminathan, Jolie Nguyen, Shivank Garg, Kevin Zhu, Vasu Sharma
| Challenge: | Fine-grained image-caption alignment is crucial for vision-language models in socially critical contexts. |
| Approach: | They present a benchmarking dataset for fine-grained image-caption alignment in safety and culture contexts. |
| Outcome: | The proposed benchmarks show that models perform better at confirming correct pairs than rejecting incorrect ones on dual alignment tasks. |
Audio Description Generation in the Era of LLMs and VLMs: A Review of Transferable Generative AI Technologies (2025.findings-naacl)
Copied to clipboard
| Challenge: | Audio descriptions (ADs) are acoustic commentaries designed to assist blind and visually impaired individuals in accessing digital media content. |
| Approach: | They examine how state-of-the-art NLP and CV technologies can be applied to generate ADs . they identify essential research directions for the future . |
| Outcome: | The proposed technologies can be applied to generate audio descriptions (ADs) the process is time-consuming and costly, and requires significant human effort . the authors identify key research directions for the future . |
MTP: A Dataset for Multi-Modal Turning Points in Casual Conversations (2024.acl-short)
Copied to clipboard
| Challenge: | a new problem setting is designed to detect critical moments in conversations . a human-annotated multi-modal dataset is used to classify and detect turning points . |
| Approach: | They propose a problem setting focusing on turning points in conversations as TPs . they propose MTPC, MTPD, & MTPR tasks to classify and detect turning points . |
| Outcome: | The proposed model achieves an F1-score of 0.88 in classification and 0.61 in detection . it uses state-of-the-art vision-language models to construct a narrative from the videos . |
World-to-Words: Grounded Open Vocabulary Acquisition through Fast Mapping in Vision-Language Models (2023.acl-long)
Copied to clipboard
| Challenge: | GOVA examines grounding and bootstrapping in open-world language learning. |
| Approach: | They propose a visually-grounded language model that uses grounding as an objective . they propose GOVA to investigate grounding and bootstrapping in open-world language learning . |
| Outcome: | The proposed model is faster and faster grounded than previous models, the authors show . they show that grounding helps the model to learn unseen words more rapidly and robustly . |
BigTokDetect: A Clinically-Informed Vision–Language Modeling Framework for Detecting Pro-Bigorexia Videos on TikTok (2026.eacl-long)
Copied to clipboard
Minh Duc Chu, Kshitij Pawar, Zihao He, Roxanna Sharifi, Ross M. Sonnenblick, Magdalayna Curry, Laura DAdamo, Lindsay Young, Stuart Murray, Kristina Lerman
| Challenge: | Social media platforms face escalating challenges in detecting harmful content that promotes muscle dysmorphic behaviors and cognitions (bigorexia). |
| Approach: | They propose a framework for detecting pro-bigorexia content on TikTok using an expert-annotated multimodal benchmark dataset of over 2,200 Tiktok videos labeled by clinical psychiatrists. |
| Outcome: | The proposed framework improves on fine-grained subcategories while commercial models achieve the highest accuracy on primary categories. |
Which Modality should I use - Text, Motif, or Image? : Understanding Graphs with Large Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Current research typically employs limited setups with small real-world graphs. |
| Approach: | They propose a new approach to encoding a graph with diverse modalities, such as text, image, and motif, coupled with prompts to approximate a diagram’s global connectivity. |
| Outcome: | The proposed approach improves performance of LLMs in graph structure analysis by focusing on homophily, motif presence, and graph difficulty. |
SIMPLOT: Enhancing Chart Question Answering by Distilling Essentials (2025.findings-naacl)
Copied to clipboard
| Challenge: | Recent advances in vision-language models have accelerated research into models capable of advanced reasoning based on images. |
| Approach: | They propose a method that leverages vision-language models to convert charts into table format . they use Large Language Model (LLM) for reasoning to extract only the essential information . |
| Outcome: | The proposed method extracts only the elements necessary for chart reasoning without the need for additional annotations or datasets. |
On the Additive Compositionality of Task Vectors in Vision–Language Models (2026.eacl-short)
Copied to clipboard
| Challenge: | In-context learning (ICL) in large language models (LLMs) has been shown to operate through task vectors, but its extension to vision-language models (VLMs) remains underexplored. |
| Approach: | They construct visual reasoning tasks with clearly defined subtasks and extract task vectors from few-shot demonstrations. |
| Outcome: | The proposed model can be extended to vision-language models (VLMs) by adding the vectors of its constituent subtasks. |
VLStereoSet: A Study of Stereotypical Bias in Pre-trained Vision-Language Models (2022.aacl-main)
Copied to clipboard
| Challenge: | Existing studies on pre-trained vision-language models have focused on measuring biases and stereotypes in a single modality. |
| Approach: | They extend a recently released stereotypical bias dataset into a vision-language probing dataset called VLStereoSet to measure stereotypical biased vision-linguistic models. |
| Outcome: | The proposed probing task measures stereotypical bias in vision-language models and its intra-modal and inter-modal biases. |
Think Before You Act: A Two-Stage Framework for Mitigating Gender Bias Towards Vision-Language Tasks (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing vision-language models focus on salient attributes but ignore contextualized nuances, resulting in gender bias. |
| Approach: | They propose a task-agnostic generation framework to mitigate gender bias in vision-language models. |
| Outcome: | The proposed framework can mitigate gender bias in vision-language models . it yields all-sided but gender-obfuscated narratives, which prevents concentration on localized image features, especially gender attributes. |
Annotation-Efficient Vision-Language Model Adaptation to the Polish Language Using the LLaVA Framework (2026.eacl-srw)
Copied to clipboard
Grzegorz Statkiewicz, Alicja Dobrzeniecka, Karolina Seweryn, Aleksandra Krasnodębska, Karolina Piosek, Katarzyna Bogusz, Sebastian Cygert, Wojciech Kusa
| Challenge: | Currently, most vision-language models are trained on English-centric data, limiting their usability for non-English-speaking users. |
| Approach: | They reproduce and adapt LLaVA-Next methodology to create Polish VLMs . they use a fully automated pipeline for translating and filtering existing multimodal datasets based on Polish data for OCR and culturally specific tasks. |
| Outcome: | The proposed model improves on a Polish-adapted model and shows higher quality captions in generative evaluations. |
VLIS: Unimodal Language Models Guide Multimodal Language Generation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing vision-language models face challenges in tasks that require complex linguistic understanding. |
| Approach: | They propose a framework that combines visual conditioning and linguistic understanding of unimodal text-only language models without further training to improve vision-language models. |
| Outcome: | The proposed framework improves vision-language models on diverse tasks including commonsense understanding and complex text generation. |
ColorBrowserAgent: Complex Long-Horizon Browser Agent with Adaptive Knowledge Evolution (2026.acl-industry)
Copied to clipboard
Jihong Wang, Jiamu Zhou, Weiming Zhang, Teng Wang, Weiwen Liu, Zhuosheng Zhang, Xingyu Lou, Weinan Zhang, Huarong Deng, Jun Wang
| Challenge: | Xue et al., 2025): deploying autonomous web agents in production remains difficult due to site heterogeneity and long-horizon instability. |
| Approach: | They propose a knowledge-evolving agent that can be used to automate web workflows . they use human-in-the-loop knowledge adaptation and knowledge-aligned progressive summarization . |
| Outcome: | Experiments on WebArena, WebChoreAren and industrial deployment show it outperforms baselines. |
Coarse-to-Fine Contrastive Learning in Image-Text-Graph Space for Improved Vision-Language Compositionality (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have highlighted severe limitations of contrastive learning models in their ability to perform compositional reasoning over objects, attributes, and relations. |
| Approach: | They propose a graph decomposition framework and negative mining techniques to improve attribute binding and relation understanding of scene graphs. |
| Outcome: | The proposed approach improves attribute binding, relation understanding, generalization, and productivity on multiple benchmarks. |
FlagEval-Arena: A Side-by-Side Comparative Evaluation Platform for Large Language Models and Text-Driven AIGC (2025.acl-demo)
Copied to clipboard
Jing-Shu Zheng, Richeng Xuan, Bowen Qin, Zheqi He, Tongshuai.ren Tongshuai.ren, Xuejing Li, Jin-Ge Yao, Xi Yang
| Challenge: | a new evaluation platform for large language models and text-driven AIGCs is available for free. |
| Approach: | They propose an evaluation platform for side-by-side comparisons of large language models and text-driven AIGC systems. |
| Outcome: | a new evaluation platform for large language models and text-driven AIGC systems is available for free . the platform is more focused on the Chinese language and more models developed by Chinese institutes . |
How Do Inpainting Artifacts Propagate to Language? (2026.acl-short)
Copied to clipboard
| Challenge: | Figure 1 shows representative examples of visual artifacts introduced by diffusion-based inpainting . despite visually plausible reconstructions, localized inpainding artifactors lead to object substitutions, attribute changes, or category-level errors in downstream captions. |
| Approach: | They propose a diagnostic setup in which masked image regions are reconstructed and then provided to captioning models. |
| Outcome: | The proposed diagnostic framework can be used to examine how visual artifacts affect language generation in vision-language models. |
A Unified Framework and Dataset for Assessing Societal Bias in Vision-Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies have highlighted the existence of social biases within large vision and language models. |
| Approach: | They propose a framework for systematically evaluating gender, race, and age biases in vision-language models with respect to professions. |
| Outcome: | The proposed framework covers all supported inference modes of the recent vision-language models, including image-to-text, text-to image, and image- to-image. |
DRISHTIKON: A Multimodal Multilingual Benchmark for Testing Language Models’ Understanding on Indian Culture (2025.emnlp-main)
Copied to clipboard
Arijit Maji, Raghvendra Kumar, Akash Ghosh, null Anushka, Nemil Shah, Abhilekh Borah, Vanshika Shah, Nishant Mishra, Sriparna Saha
| Challenge: | DRISHTIKON is a first-of-its-kind multimodal and multilingual benchmark centered exclusively on Indian culture. |
| Approach: | They evaluate a wide range of vision-language models across zero-shot and chain-of-thought settings and use them to evaluate cultural understanding of generative AI systems. |
| Outcome: | The DRISHTIKON dataset covers 15 languages, all states and union territories, and incorporating over 64,000 aligned text-image pairs. |
Colorism in Multimodal AI: An Empirical Exploration of Socioeconomic Linguistic Bias in Text-to-Image Generation (2026.eacl-srw)
Copied to clipboard
| Challenge: | Socioeconomic inequalities worldwide are deeply linked to ethnoracial hierarchies and stereotypes, argues a new study. |
| Approach: | They use a Monk Skin Tone scale to benchmark VLMs and annotators . they then use linguistic cues to vary skin-tone representations in text-to-image generation . |
| Outcome: | The study compares 3 small VLMs and 60 human annotators on the monk skin tone scale with 210 occupations and produces over 2,500 portraits across 3 large VLM models. |
From Grounding to Manipulation: Case Studies of Foundation Model Integration in Embodied Robotic Systems (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a new study examines the operational characteristics of different integration strategies for robotics . end-to-end vision-language-action models implicitly unify perception and planning . |
| Approach: | They propose end-to-end vision-language-action models that implicitly unify perception and planning . they also propose modular pipelines using either vision-linguistic models or MLLMs . |
| Outcome: | The proposed frameworks implicitly unify perception and planning, and modular pipelines using either vision-language models or multimodal large language models. |
Seeing is Believing, but How Much? A Comprehensive Analysis of Verbalized Calibration in Vision-Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to quantify uncertainty are limited in vision-language models . however, current models display notable miscalibration across diverse tasks and settings . |
| Approach: | They evaluate verbalized confidence in vision-language models using visual reasoning . they propose a prompting strategy that improves confidence alignment in multimodal settings . |
| Outcome: | The proposed method improves confidence alignment across multimodal settings. |
From Wrong To Right: A Recursive Approach Towards Vision-Language Explanation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for generating insightful explanations with limited annotations are limited. |
| Approach: | They propose a method that iteratively computes visual features, an answer, and an explanation to improve the explanation quality step by step until the answer converges. |
| Outcome: | The proposed method outperforms previous methods while utilizing 5% of the human-annotated explanations across 10 metrics, showing up to 4.2 and 1.3 increases in BLEU-1 score on the VCR and VQA-X datasets. |
StarFlow: Generating Structured Workflow Outputs From Sketch Images (2026.eacl-long)
Copied to clipboard
Patrice Bechard, Chao Wang, Amirhossein Abaskohi, Juan A. Rodriguez, Christopher Pal, David Vazquez, Spandana Gella, Sai Rajeswar, Perouz Taslakian
| Challenge: | Despite being widely used, building workflows can be complex, often requiring manual configuration through low-code platforms or visual programming tools. |
| Approach: | They propose a framework for generating structured workflow outputs from sketches using vision-language models to automate the process. |
| Outcome: | The proposed framework outperforms large vision-language models in the task of generating structured workflow outputs from sketches and diagrams. |
One More Modality: Does Abstract Meaning Representation Benefit Visual Question Answering? (2025.findings-emnlp)
Copied to clipboard
| Challenge: | incorporating explicit semantic information, in the form of Abstract Meaning Representation graphs, can enhance VQA models. |
| Approach: | They augment two vision-language models with sentence- and document-level AMRs . they find that in well-resourced settings, models are negatively impacted by AMR . |
| Outcome: | The proposed model improves in well-resourced and low-resource settings with AMR graphs . the model achieves 13.1% relative gain using sentence-level AMRs compared with the smaller model . |
Unified Multimodal Interleaved Document Representation for Retrieval (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing methods focus on textual content, ignoring the fact that documents can contain multiple modalities. |
| Approach: | They propose a method that holistically embeds documents interleaved with multiple modalities . they use vision-language models that combine text, images, and tables into a unified format . |
| Outcome: | The proposed method outperforms baselines on textual and multimodal queries. |
A Computational Approach to Visual Metonymy (2026.eacl-long)
Copied to clipboard
| Challenge: | Visual metonymy is a form of indirect representation in which an image evokes a concept not by depicting it directly, but by presenting visually associated cues that invite the viewer to infer the intended meaning. |
| Approach: | They propose a pipeline grounded in semiotic theory that leverages large language models and text-to-image models to generate metonymic visual representations. |
| Outcome: | The proposed pipeline exploits large language models and text-to-image models to generate metonymic visual representations. |
Data or Language Supervision: What Makes CLIP Better than DINO? (2025.findings-emnlp)
Copied to clipboard
| Challenge: | CLIP outperforms self-supervised models like DINO as vision encoders for vision-language models (VLMs) but it remains unclear whether this advantage stems from CLIP’s language supervision or its much larger training data. |
| Approach: | Embedding analysis shows CLIP captures high-level semantics while DINO is more responsive to low-level features like colors and styles. |
| Outcome: | Embedding analysis shows that CLIP captures high-level semantics, while DINO is more responsive to low-level features like colors and styles. |
Enhancing Fine-Grained Image Classifications via Cascaded Vision Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Fine-grained image classification is a challenge for vision-language models (VLMs) such as CLIP, which struggle to distinguish between semantically similar classes due to insufficient supervision for fine-grain tasks. |
| Approach: | They propose a framework that harnesses the complementary strengths of both CLIP-like and LVLMs to tackle these challenges. |
| Outcome: | The proposed framework outperforms existing models on multiple fine-grained datasets, particularly the Stanford Cars dataset. |
Evaluation of Multilingual Ability to Use Spatial Deictic Expressions in Vision-Language Models (2026.acl-srw)
Copied to clipboard
| Challenge: | Existing studies have focused on the ability of vision-language models to utilize spatial deictic expressions, which depend on the situation of utterance. |
| Approach: | They develop a benchmark to evaluate the multilingual ability of VLMs to use spatial deictic expressions in four languages. |
| Outcome: | The proposed models use demonstratives in a different manner from humans, particularly in selecting demonstrative based on distance from the object. |
Intrinsic Bias is Predicted by Pretraining Data and Correlates with Downstream Performance in Vision-Language Encoders (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent work has found that vision-language models trained under the Contrastive Language Image Pre-training framework contain intrinsic social biases, but how these biase relates to downstream performance has been unclear. |
| Approach: | They present the largest comprehensive analysis to-date of how upstream pre-training factors and downstream performance of CLIP models relate to their intrinsic biases. |
| Outcome: | The proposed model performance analysis shows that the choice of pre-training dataset is the most significant upstream predictor of bias, whereas architectural variations have minimal impact. |
Finding Culture-Sensitive Neurons in Vision-Language Models (2026.eacl-long)
Copied to clipboard
| Challenge: | Vision-language models struggle on culturally situated inputs, study shows . despite impressive performance, many VLMs struggle on such culturally grounded inputs . |
| Approach: | They propose a new margin-based selector to identify neurons associated with cultural selectivity . they also introduce a model-dependent decoder to identify such neurons . |
| Outcome: | The proposed model outperforms probability- and entropy-based methods in identifying neurons associated with cultural selectivity. |
Retrieving Multimodal Prompts for Generative Visual Question Answering (2023.findings-acl)
Copied to clipboard
| Challenge: | Visual question answering (VQA) is a multimodal machine learning problem that challenges a model to answer a question posed about an image. |
| Approach: | They propose a generative model enhanced by multimodal prompt retrieval that integrates retrieved prompts and multimodal features to generate answers in free text. |
| Outcome: | The proposed model outperforms its non-retrieval counterpart by 30% on medical VQA tasks. |
VSCBench: Bridging the Gap in Vision-Language Model Safety Calibration (2025.findings-acl)
Copied to clipboard
Jiahui Geng, Qing Li, Zongxiong Chen, Yuxia Wang, Derui Zhu, Zhuohan Xie, Chenyang Lyu, Xiuying Chen, Preslav Nakov, Fakhri Karray
| Challenge: | Existing safety calibration methods focus on model undersafety, where the model responds to hazardous queries, while neglecting oversafetiness, where models refuse to answer safe queries. |
| Approach: | They propose safety calibration which addresses both undersafety and oversafetiness by comparing model responses to a novel dataset of 3,600 image-text pairs. |
| Outcome: | The proposed methods have been used to evaluate safety calibration across image-centric and text-centric scenarios. |
Emo3D: Metric and Benchmarking Dataset for 3D Facial Expression Generation from Emotion Description (2025.findings-naacl)
Copied to clipboard
Mahshid Dehghani, Amirahmad Shafiee, Ali Shafiei, Neda Fallah, Farahmand Alizadeh, Mohammad Mehdi Gholinejad, Hamid Behroozi, Jafar Habibi, Ehsaneddin Asgari
| Challenge: | Existing 3D facial emotion modeling models are constrained by limited emotion classes and insufficient datasets. |
| Approach: | They propose a 3D facial emotion modeling dataset that spans a wide spectrum of human emotions . they use large language models to generate a diverse array of textual descriptions . |
| Outcome: | Emo3D is an extensive dataset that spans human emotions with images and 3D blendshapes. |
DRIVINGVQA: A Dataset for Interleaved Visual Chain-of-Thought in Real-World Driving Scenarios (2026.findings-eacl)
Copied to clipboard
| Challenge: | Chain-of-thought (CoT) prompting is a prompting strategy that improves reasoning in large language models, but its effectiveness in vision-language models remains limited due to over-reliance on textual cues and memorized knowledge. |
| Approach: | They propose a visual question-answering dataset derived from driving theory exams that incorporates textual explanations with visual tokens extracted from entities relevant to the reasoning process. |
| Outcome: | The proposed approach outperforms chain-of-thought prompting in large language models and vision-language models in real-world scenarios. |
VISTA: Visualized Text Embedding For Universal Multi-Modal Retrieval (2024.acl-long)
Copied to clipboard
| Challenge: | Existing methods for multimodal retrieval are mostly text-oriented, which lack the capability to process visual information. |
| Approach: | They propose a multi-modal multi-text embedding model VISTA which extends a powerful text encoder with the image understanding capability by introducing visual token embedds. |
| Outcome: | The proposed model achieves superior performance across a variety of multi-modal retrieval tasks in zero-shot and supervised settings. |
Beyond End-to-End VLMs: Leveraging Intermediate Text Representations for Superior Flowchart Understanding (2025.naacl-long)
Copied to clipboard
| Challenge: | Flowcharts are typically presented as images, driving the trend of using vision-language models for end-to-end flowchart understanding. |
| Approach: | They propose a vision-language model (VLM) that generates textual representations from flowchart images and a textual Reasoner that performs question-answering based on the text representations. |
| Outcome: | Experiments on the FlowVQA and FlowLearn benchmarks demonstrate TextFlow’s state-of-the-art performance as well as its robustness. |
MatViX: Multimodal Information Extraction from Visually Rich Articles (2025.naacl-long)
Copied to clipboard
Ghazal Khalighinejad, Sharon Scott, Ollie Liu, Kelly L. Anderson, Rickard Stureborg, Aman Tyagi, Bhuwan Dhingra
| Challenge: | Existing methods for multimodal information extraction are limited due to the multimodal nature of scientific articles and complex interconnections between data points. |
| Approach: | They propose a benchmark to extract structured information from scientific articles . they use curated JSON files extracted from text, tables, and figures . |
| Outcome: | The proposed benchmark is based on 324 full-length research articles and 1,688 complex structured JSON files curated by experts in polymer nanocomposites and biodegradation. |
TurkingBench: A Challenge Benchmark for Web Agents (2025.naacl-long)
Copied to clipboard
Kevin Xu, Yeganeh Kordi, Tanay Nayak, Adi Asija, Yizhong Wang, Kate Sanders, Adam Byerly, Jingyu Zhang, Benjamin Van Durme, Daniel Khashabi
| Challenge: | TurkingBench is a benchmark consisting of tasks presented as web pages with textual instructions and multi-modal contexts. |
| Approach: | They propose to use HTML pages to perform various annotation tasks on crowdsourcing platforms. |
| Outcome: | The proposed model outperforms other models on the TurkingBench benchmark. |
Video Question Answering with Phrases via Semantic Roles (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing VidQA evaluation metrics limit the models’ application scenario to a single-word answer or selecting a phrase from a fixed set of phrases. |
| Approach: | They propose to leverage video descriptions to mask out certain phrases to enable evaluation of answer phrases. |
| Outcome: | The proposed model reduces the influence of language bias on VidQA datasets by retrieving a video having a different answer for the same question. |
Learning Mutually Informed Representations for Characters and Subwords (2024.findings-naacl)
Copied to clipboard
| Challenge: | Pretrained language models rely on subword tokenization to process text as a sequence of subwords. |
| Approach: | They propose a character-subword language model that integrates character and subword modalities into one model. |
| Outcome: | The proposed model outperforms its backbone language models on English sequence labeling and classification tasks. |
BLEnD-Vis: Benchmarking Multimodal Cultural Understanding in Vision Language Models (2026.eacl-long)
Copied to clipboard
Bryan Chen Zhengyu Tan, Weihua Zheng, Zhengyuan Liu, Nancy F. Chen, Hwaran Lee, Kenny Tsu Wei Choo, Roy Ka-Wei Lee
| Challenge: | Existing evaluations assess static recall or isolated visual grounding, leaving unanswered whether VLMs possess robust and transferable cultural understanding. |
| Approach: | They propose a multimodal, multicultural benchmark to evaluate the robustness of everyday cultural knowledge in vision-language models across linguistic rephrasings and visual modalities. |
| Outcome: | ‘BLEnD-Vis‘ constructs 313 culturally grounded question templates spanning 16 regions and generates three aligned multiple-choice formats. |
Vision Language Model Helps Private Information De-Identification in Vision Data (2025.findings-acl)
Copied to clipboard
| Challenge: | Visual Language Models (VLMs) have gained popularity due to their ability to solve imagerelated tasks. |
| Approach: | They propose a framework to enhance privacy awareness of visual language models . they use a specialized instruction-tuning dataset and a tailored training methodology . |
| Outcome: | The proposed framework outperforms existing approaches in handling private information. |
Synonym relations affect object detection learned on vision-language data (2024.findings-naacl)
Copied to clipboard
| Challenge: | a recent study shows that vision-language models that accept textual input are not robust to variations in how input is provided. |
| Approach: | They propose two approaches to improve vision-language object detectors' performance . they use back-translation and class embedding enrichment to improve their models . |
| Outcome: | The proposed approaches improve performance on synonyms from mAP@0.3=33.87% to 37.93%. |
‘Just because you are right, doesn’t mean I am wrong’: Overcoming a bottleneck in development and evaluation of Open-Ended VQA tasks (2021.eacl-main)
Copied to clipboard
Man Luo, Shailaja Keyur Sampat, Riley Tallman, Yankai Zeng, Manuha Vancha, Akarshan Sajja, Chitta Baral
| Challenge: | Existing visual question answering datasets assume only one ground truth answer for each question. |
| Approach: | They propose alternative answer sets (AAS) of ground-truth answers to address this limitation . they modify top VQA solvers to support multiple plausible answers for a question . |
| Outcome: | The proposed approach improves on the GQA dataset and shows that it is more efficient than previous approaches. |
Your Vision-Language Model Itself Is a Strong Filter: Towards High-Quality Instruction Tuning with Data Selection (2024.findings-acl)
Copied to clipboard
Ruibo Chen, Yihan Wu, Lichang Chen, Guodong Liu, Qi He, Tianyi Xiong, Chenxi Liu, Junfeng Guo, Heng Huang
| Challenge: | Existing data selection methods for instruction-following large language models rely on unreliable scores or use downstream tasks for selection. |
| Approach: | They propose a method that utilizes the VLM itself as a filter to select high-quality instruction-tuning data. |
| Outcome: | The proposed method can reach better results compared to full data settings with merely about 15% samples and can achieve superior performance against competitive baselines. |
Multimodal Cognitive Reframing Therapy via Multi-hop Psychotherapeutic Reasoning (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing studies have focused on text-based cognitive reframing, but neglected the importance of non-verbal evidence in real-life therapy. |
| Approach: | They propose a dataset that pairs each GPT-4-generated dialogue with an image that reflects the virtual client’s facial expressions to better mirror real psychotherapy, where facial expression leads to interpreting implicit emotional evidence. |
| Outcome: | The proposed approach outperforms existing methods with LLMs and vision-language models and provides more thoughtful and empathetic suggestions. |
What’s Missing in Vision-Language Models? Probing Their Struggles with Causal Order Reasoning (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing benchmarks often include a mixture of reasoning questions, making it difficult to truly assess VLMs’ causal reasoning abilities. |
| Approach: | They propose two new benchmarks specifically designed to isolate and rigorously evaluate VLMs’ causal reasoning abilities. |
| Outcome: | The proposed benchmarks show that vision-language models perform poorly on causal reasoning tasks, often only marginally surpassing random guessing. |
From the Least to the Most: Building a Plug-and-Play Visual Reasoner via Data Synthesis (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing models consisting of multiple steps of visual and language processing are limited in the visual and visual processing community . a visual reasoner is a plug-and-play approach that can be used to improve VLMs' reasoning abilities. |
| Approach: | They propose a least-to-most visual reasoning paradigm that divides a question into sub-questions and invokes external tools for resolving sub-questions. |
| Outcome: | The proposed method can improve four VLMs on four VQA benchmarks. |
CLEVR_HYP: A Challenge Dataset and Baselines for Visual Question Answering with Hypothetical Actions over Images (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing research on visual question answering is limited to information explicitly present in an image or a video. |
| Approach: | They propose a vision-language question answering task based on a CLEVR dataset . they modify existing methods and propose baseline solvers for this task . |
| Outcome: | The proposed model motivates the development of better vision-language models . it provides insights about the capability of diverse architectures to perform joint reasoning over image-text modality. |
Cross-modal Attention Congruence Regularization for Vision-Language Relation Alignment (2023.acl-long)
Copied to clipboard
| Challenge: | Despite recent progress towards scaling up multimodal vision-language models, these models struggle on compositional generalization benchmarks such as Winoground. |
| Approach: | They propose to use a cross-modal attention regularization loss to enforce relation alignment by capturing the semantic relation ‘in’ to match the visual attention from the mug to the grass. |
| Outcome: | The proposed approach improves Winoground Group score by 5.75 points . |
Visually-Guided Policy Optimization for Multimodal Reasoning (2026.acl-long)
Copied to clipboard
| Challenge: | Existing RLVRs lack visual faithfulness due to text-dominated reasoning . a novel framework to reinforce visual focus during policy optimization is proposed . |
| Approach: | They propose a framework to reinforce visual focus during policy optimization using visual attention compensation mechanism. |
| Outcome: | The proposed framework exhibits better visual activation and superior performance in multimodal reasoning and visual-dependent tasks. |
GOBench: Stage-Wise Diagnostics and the Visual Paradox in Multimodal Graph Optimization (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks fail to represent multimodal problem specifications, score outcomes only and cannot localize where failures occur along the modeling pipeline. |
| Approach: | They propose a Graph Optimization benchmark that aligns multiple modalities with solver-derived oracles and a diagnostic protocol that evaluates intermediate artifacts as well as end results. |
| Outcome: | Graph Optimization benchmark (GOBench) evaluates intermediate artifacts as well as end results . vision reliably increases inference cost, while reliability impact is regime-dependent . current benchmarks fail to represent multimodal problem specifications, fail to localize failures . |
Instruction Tuning with and without Context: Behavioral Shifts and Downstream Impact (2026.eacl-long)
Copied to clipboard
Hyunji Lee, Seunghyun Yoon, Yunjae Won, Hanseok Oh, Geewook Kim, Trung Bui, Franck Dernoncourt, Elias Stengel-Eskin, Mohit Bansal, Minjoon Seo
| Challenge: | Prior work on instruction tuning datasets combined these data types without examining their distinct effects. |
| Approach: | They investigate how training LLMs with or without context affects model behavior and performance . they find that using context-augmented data as the backbone for vision-language models reduces hallucination . |
| Outcome: | The proposed training with context-augmented data reduces hallucination and improves grounding in the visual domain. |
VizoMem: A Visual-Textual Memory Framework for Efficient Long-Horizon Reasoning (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing systems that use long-context modeling incur computational and memory overhead. |
| Approach: | They propose a visual memory framework that pre-rendered text into structured images and stored as visual notes for agentic systems. |
| Outcome: | The proposed system reduces token consumption while preserving effective long-term memory recall. |
VLM2-Bench: A Closer Look at How Well VLMs Implicitly Link Explicit Matching Visual Cues (2025.acl-long)
Copied to clipboard
| Challenge: | Existing vision-language models lack the ability to visually link matching visual cues across images or frames. |
| Approach: | They propose a benchmark to assess whether vision-language models can Visually Link Matching cues with 9 subtasks and over 3,000 test cases. |
| Outcome: | The proposed benchmarks on multiple images and videos do not demonstrate that vision-language models can link visual cues across images or frames. |
From Local Concepts to Universals: Evaluating the Multicultural Understanding of Vision-Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Vision-Language Models (VLMs) have shown emerging capabilities through large-scale training that have made them gain popularity in recent years. |
| Approach: | They propose to perform retrieval across universals and cultural visual grounding tasks to assess cultural diversity across universal and culture-specific local concepts. |
| Outcome: | The proposed benchmarks show that the models perform significantly across cultures, underscoring the need for enhancing multicultural understanding in vision-language models. |
Does Chain-of-Thought Reasoning Help Mobile GUI Agents? An Empirical Study (2026.findings-acl)
Copied to clipboard
| Challenge: | Reasoning capabilities have improved vision-language models in domains like math, coding, and visual question-answering, but their impact on real-world applications remains unclear. |
| Approach: | They evaluate six pairs of VLMs by comparing their base and reasoning-enhanced versions across static and interactive benchmarks. |
| Outcome: | The reasoning-enhanced models perform better on static and interactive benchmarks than non-reasoning models. |
Asymmetric Relational-Geometry Driven Universal Adversarial Perturbations for Vision-Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing universal adversarial perturbation (UAP) methods suffer from limited cross-model transferability in black-box scenarios. |
| Approach: | They propose an optimization-based framework that learns universal perturbations under an asymmetric relational-geometry driven objective. |
| Outcome: | The proposed framework outperforms state-of-the-art models in black-box transfer settings. |
VisPCO: Visual Token Pruning Configuration Optimization via Budget-Aware Pareto-Frontier Learning for Vision-Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for visual token pruning rely on predefined configurations without determining whether they achieve optimal performance. |
| Approach: | They propose a framework that formulates visual token pruning as a Pareto configuration optimization problem to automatically identify optimal configurations. |
| Outcome: | The proposed framework approximates the empirical Pareto frontier obtained through grid search and generalizes well across pruning methods and VLM architectures. |
RAP: Efficient Text-Video Retrieval with Sparse-and-Correlated Adapter (2024.findings-acl)
Copied to clipboard
Meng Cao, Haoran Tang, Jinfa Huang, Peng Jin, Can Zhang, Ruyang Liu, Long Chen, Xiaodan Liang, Li Yuan, Ge Li
| Challenge: | Text-Video Retrieval (TVR) aims to align relevant video content with natural language queries. |
| Approach: | They propose to conduct efficient text-video Retrieval with a salient-and-correlated AdaPter . they propose a low-rank modulation module to refine per-image features from frozen CLIP backbone . |
| Outcome: | Experiments on four TVR datasets show that the proposed method performs better than other methods. |
ChartAssistant: A Universal Chart Multimodal Language Model via Chart-to-Table Pre-training and Multitask Instruction Tuning (2024.findings-acl)
Copied to clipboard
| Challenge: | Charts are an effective tool for understanding data patterns, but their combination of graphical elements and textual components poses challenges for general-purpose multimodal models. |
| Approach: | They propose a chart-based vision-language model for universal chart comprehension and reasoning that leverages a dataset of chart-related tasks. |
| Outcome: | The proposed model outperforms the state-of-the-art charts with zero-shot setting on various chart tasks. |
Look Again, Think Slowly: Enhancing Visual Reflection in Vision-Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in text-only "slow thinking" reasoning have prompted efforts to transfer this capability to vision-language models (VLMs). |
| Approach: | They propose a VRM Reflection-V which enhances visual reflection based on reasoning data for cold-start and reward design for reinforcement learning. |
| Outcome: | The proposed model improves visual reflection for cold-start and reward design for reinforcement learning (RL) it maintains a stronger and more consistent reliance on visual information during visual reasoning, indicating effective enhancement in visual reflection capabilities. |
Position Really Matters: Towards a Holistic Approach for Prompt Tuning (2025.findings-naacl)
Copied to clipboard
| Challenge: | Prompt tuning is effective in extracting knowledge from foundation models, but its effectiveness is uncertain. |
| Approach: | They propose a parametric prompt tuning strategy that dynamically determines different factors of prompts based on specific tasks or instances. |
| Outcome: | The proposed approach improves performance across a wide range of tasks including NLP, vision recognition, and vision-language tasks. |
TCPO: Thought-Centric Preference Optimization for Effective Embodied Decision-making (2025.emnlp-main)
Copied to clipboard
Kechen Jiao, Zhirui Fang, Jiahao Liu, Bei Li, Qifan Wang, Xinyu Liu, Junhao Ruan, Zhongjian Qiao, Yifan Zhu, Yaxin Xu, Jingang Wang, Xiu Li
| Challenge: | Existing post-SFT methods for embodied AI are constrained by sparse rewards and action-only optimization, resulting in low sample efficiency, poor consistency, and model degradation. |
| Approach: | They propose to integrate Thought-Centric Preference Optimization (TCPO) into embodied decision-making by transforming sparse reward signals into richer step sample pairs. |
| Outcome: | The proposed approach achieves an average success rate of 26.67% in the ALFWorld environment, and a 6% improvement over RL4VLM. |
Multi3Hate: Multimodal, Multilingual, and Multicultural Hate Speech Detection with Vision–Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | a new study shows that cultural background significantly affects multimodal hate speech moderation models . a limited dataset excludes multi-modal forms of hate and excludes non-English-speaking cultures . the lowest pairwise label agreement between the USA and India is due to cultural factors . |
| Approach: | They use a multimodal and multilingual parallel hate speech dataset to examine cultural differences . they find that cultural background significantly affects multimodal hate speech annotation . |
| Outcome: | The proposed dataset shows that cultural background significantly affects multimodal hate speech annotation. |
AI Knows Where You Are: Exposure, Bias, and Inference in Multimodal Geolocation with KoreaGEO (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing benchmarks show coarse granularity, linguistic bias, and a neglect of multimodal privacy risks. |
| Approach: | They propose a benchmark for visual-language models that analyzes social photos to assess location privacy risks. |
| Outcome: | The proposed benchmarks show coarse granularity, linguistic bias, and neglect of privacy risks. |
Towards Mitigating Modality Bias in Vision-Language Models for Temporal Action Localization (2026.acl-long)
Copied to clipboard
| Challenge: | Existing vision-language models overemphasize linguistic priors, leading to modality bias. |
| Approach: | They propose a vision-language aggregation framework that mitigates modality bias in TAL by preserving vision as the dominant signal while adaptively exploiting language only when beneficial. |
| Outcome: | Experiments on THUMOS14 show that the proposed model outperforms state-of-the-art models by up to 3.2% mAP. |
AutoGUI: Scaling GUI Grounding with Automatic Functionality Annotations from LLMs (2025.acl-long)
Copied to clipboard
| Challenge: | Existing datasets for UI-VLMs contain large-scale context-free element annotations or contextualized functional descriptions for elements at a small scale. |
| Approach: | They propose an auto-annotation pipeline that generates massive UI element functionality annotations based on UI content changes induced by interacting with the elements. |
| Outcome: | The proposed pipeline improves accuracy and scales well with human evaluation of a high-quality AutoGUI-704k dataset. |
Bridging the Visual Gap: Fine-Tuning Multimodal Models with Knowledge-Adapted Captions (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent work focuses on training vision-language models with long, detailed image captions, but small-scale VLMs struggle to balance the richness of these captions with the risk of hallucinations. |
| Approach: | They propose an evaluation framework that breaks down generated captions into individual propositions, assessing each in isolation. |
| Outcome: | The proposed framework outperforms baselines in both automatic metrics and human evaluations on small-scale vision-language models with long, detailed captions. |
DesignCLIP: Multimodal Learning with CLIP for Design Patent Understanding (2025.findings-emnlp)
Copied to clipboard
| Challenge: | patent images often lack comprehensive visual context and semantic information, authors say . recent advances in vision-language models offer promising opportunities for patent analysis . |
| Approach: | They develop a framework for design patent analysis using large-scale patent dataset . they validate the effectiveness of DesignCLIP across various downstream tasks . |
| Outcome: | The proposed framework outperforms baseline and SOTA models on all tasks. |
LATTE: Learning to Think with Vision Specialists (2025.emnlp-main)
Copied to clipboard
Zixian Ma, Jianguo Zhang, Zhiwei Liu, Jieyu Zhang, Juntao Tan, Manli Shu, Juan Carlos Niebles, Shelby Heinecke, Huan Wang, Caiming Xiong, Ranjay Krishna, Silvio Savarese
| Challenge: | Open-source vision-language models excel on simple question-answering tasks, but struggle with complex questions that require both perception and reasoning. |
| Approach: | They propose a family of vision-language models that have LeArned to Think wiTh vision spEcialists by offloading perception to state-of-the-art vision models. |
| Outcome: | The proposed model achieves 4-5% gains over baselines across 6 benchmarks covering both perception and reasoning abilities. |
SPHERE: Unveiling Spatial Blind Spots in Vision-Language Models Through Hierarchical Evaluation (2025.acl-long)
Copied to clipboard
Wenyu Zhang, Wei En Ng, Lixin Ma, Yuwen Wang, Junqi Zhao, Allison Koenecke, Boyang Li, Wanglu Wanglu
| Challenge: | Current vision-language models lack multi-dimensional spatial reasoning capabilities for human-like understanding and applications. |
| Approach: | They propose a hierarchical evaluation framework that probes models across increasing levels of complexity and integrates spatial, visual, and logical understanding. |
| Outcome: | The proposed framework probes models across increasing levels of complexity, from basic skills to multi-skill integration and high-level reasoning that combines spatial, visual, and logical understanding. |
Can Textual Unlearning Solve Cross-Modality Safety Alignment? (2024.findings-emnlp)
Copied to clipboard
Trishna Chakraborty, Erfan Shayegani, Zikui Cai, Nael Abu-Ghazaleh, M. Salman Asif, Yue Dong, Amit Roy-Chowdhury, Chengyu Song
| Challenge: | integrating new modalities into large language models creates new attack surface . existing safety training techniques like SFT and RLHF are not feasible in multi-modal settings . |
| Approach: | They explore whether unlearning in the textual domain can be effective for cross-modality safety alignment. |
| Outcome: | The proposed approach reduces the Attack Success Rate (ASR) to less than 8% and preserves the utility. |
Guiding Medical Vision-Language Models with Diverse Visual Prompts: Framework Design and Comprehensive Exploration of Prompt Variations (2025.naacl-long)
Copied to clipboard
| Challenge: | Current vision-language models lack the ability to focus on specific areas designated by humans . a new framework that integrates medical entity extraction, visual prompt generation, and dataset adaptation is proposed to improve visual prompt-guided fine-tuning. |
| Approach: | They propose to use visual prompts to guide and enhance formation of region-specific attention. |
| Outcome: | The proposed framework outperforms state-of-the-art large vision-language models on medical datasets. |
Beyond Cross-Modal Alignment: Measuring and Leveraging Modality Gap in Vision-Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | a recent study shows that vision-language models have modality gaps that persist even in well-aligned models. |
| Approach: | They propose a modality-dominance score to measure and leverage modality gaps . they propose automatic interpretability metrics to evaluate these features in a scalable manner . |
| Outcome: | The proposed framework allows for training-free probing and editing methods for understanding model perception across genders and generating adversarial examples. |
Can you SPLICE it together? A Human Curated Benchmark for Probing Visual Reasoning in VLMs (2025.findings-emnlp)
Copied to clipboard
Mohamad Ballout, Okajevo Wilfred, Seyedalireza Yaghoubi, Nohayr Muhammad Abdelmoneim, Julius Mayer, Elia Bruni
| Challenge: | SPLICE is a benchmark designed to probe event-based reasoning across multiple dimensions. |
| Approach: | They introduce a human-curated benchmark to probe event-based reasoning across multiple dimensions. |
| Outcome: | The proposed benchmark includes 3,381 human-filtered videos spanning 12 categories and 180 sub-categories . results show that state-of-the-art vision-language models struggle to match human performance . |
Current Agents Fail to Leverage World Model as Tool for Foresight (2026.acl-long)
Copied to clipboard
Cheng Qian, Emre Can Acikgoz, Bingxuan Li, Xiusi Chen, Yuji Zhang, Bingxiang He, Qinyu Luo, Gokhan Tur, Dilek Hakkani-Tür, Yunzhu Li, Heng Ji
| Challenge: | Generative world models could be used to enhance agents' cognition . agents are expected to operate in settings where tasks unfold over long horizons and involve intricate chains of interdependent decisions. |
| Approach: | They propose to use vision-language models as external simulators to enhance cognition . they find that agents rarely invoke simulation and misuse predicted rollouts . |
| Outcome: | The proposed model could be used to predict future states rather than short-horizon reasoning . the model could also be used for real-world planning and robotics . |
Curr-ReFT: Overcoming Training Bottlenecks in Small-scale Vision-Language Models via Curriculum Reinforcement Finetuning (2025.findings-emnlp)
Copied to clipboard
| Challenge: | State-of-the-art vision-language models require massive scaling that limits practical deployment. |
| Approach: | They propose to use supervised fine-tuning to train small-scale vision-language models but face out-of-domain collapse when trained with traditional supervised learning (SFT). |
| Outcome: | Experiments show that curr-reFT achieves state-of-the-art performance across visual tasks in both in- and out-of domain settings and benchmarks. |
MASSV: Multimodal Adaptation and Self-Data Distillation for Speculative Decoding of Vision-Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Speculative decoding of vision-language models provides a novel way to accelerate language model inference by enabling a lightweight draft model to propose multiple tokens that a larger target model verifies simultaneously. |
| Approach: | They propose a technique that allows a lightweight draft model to propose multiple tokens that a larger target model verifies simultaneously. |
| Outcome: | The proposed technique increases accepted length by 30% and delivers speedups of up to 1.46x compared to conventional text-only drafting baselines on visually-grounded tasks. |
For-Value: Efficient Forward-Only Data Valuation for finetuning LLMs and VLMs (2026.acl-long)
Copied to clipboard
Wenlong Deng, Qi Zeng, Jiaming Zhang, Minghui Chen, Zixin Ding, Christos Thrampoulidis, Boying Gong, Xiaoxiao Li
| Challenge: | Existing methods for data valuation rely on gradient computations, making them prohibitive for billion-parameter models. |
| Approach: | They propose a forward-only data valuation framework that enables efficient batch-scalable value estimation while maintaining effectiveness. |
| Outcome: | The proposed framework matches or outperforms gradient-based baselines in detecting influential data and mislabeled data while achieving significant efficiency improvements. |
Medical Adaptation of Large Language and Vision-Language Models: Are We Making Progress? (2024.emnlp-main)
Copied to clipboard
| Challenge: | Several studies claim that domain-adaptive pretraining improves performance on downstream medical tasks. |
| Approach: | They compare medical LLMs and VLMs against their corresponding base models . they find that medical Lms outperform their base models in 12.1% of cases . |
| Outcome: | The proposed models outperform their base models on medical questions and tasks in 12.1% of cases and reach a tie in 49.8% of cases. |
ROME: Evaluating Pre-trained Vision-Language Models on Reasoning beyond Visual Common Sense (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a vision-language model with commonsense knowledge can reason beyond common sense . however, pre-trained vision-linguistic models are incapable of interpreting counter-intuitive content . |
| Approach: | They introduce a probing dataset to evaluate vision-language models' reasoning abilities . they use images that defy commonsense knowledge to test their reasoning abilities. |
| Outcome: | The proposed dataset evaluates whether pre-trained vision-language models can reason beyond common sense . it contains images that defy commonsense knowledge with regards to color, shape, material, size and position . |
DocMMIR: A Framework for Document Multi-modal Information Retrieval (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing multi-modal information retrieval models lack a comprehensive exploration of document-level retrieval . existing models suffer from the absence of cross-domain datasets at this granularity. |
| Approach: | They propose a multi-modal document retrieval framework to unify diverse document formats and domains with a comprehensive retrieval scenario. |
| Outcome: | The proposed framework improves document retrieval performance on a large multimodal dataset. |
VIGNETTE: Socially Grounded Bias Evaluation for Vision-Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Existing studies on VLM bias focus on portrait-style images and gender-occupation associations . existing studies ignore broader and more complex social stereotypes and their implied harm . |
| Approach: | They propose a large-scale VQA benchmark for evaluating bias in vision-language models . they use a question-answering framework that spans factuality, perception, stereotyping, and decision making . |
| Outcome: | The proposed framework examines bias in vision-language models using 30M+ images . findings reveal subtle, multifaceted, and surprising stereotypical patterns . |
MatCha: Enhancing Visual Language Pretraining with Math Reasoning and Chart Derendering (2023.acl-long)
Copied to clipboard
Fangyu Liu, Francesco Piccinno, Syrine Krichene, Chenxi Pang, Kenton Lee, Mandar Joshi, Yasemin Altun, Nigel Collier, Julian Eisenschlos
| Challenge: | Visual language models that are pretraining on natural images or image-text pairs crawled from the web perform poorly on visual language tasks such as ChartQA and ChartQA. |
| Approach: | They propose to perform several pretraining tasks that cover plot deconstruction and numerical reasoning which are key capabilities in visual language modeling. |
| Outcome: | The proposed model outperforms state-of-the-art methods on benchmarks such as PlotQA and ChartQA by as much as 20%. |
Enhancing Textbooks with Visuals from the Web for Improved Learning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Textbooks lack visuals that support student learning, but many lack them . e-textbooks lack such visuals, and many lack these visuals . |
| Approach: | They propose to use vision-language models to automatically enhance textbooks with images from the web. |
| Outcome: | The proposed model improves textbooks with images from the web while allowing for better pedagogical value. |
MMSciBench: Benchmarking Language Models on Chinese Multimodal Scientific Problems (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing scientific benchmarks lack human-annotated difficulty levels and structured taxonomies of scientific concepts. |
| Approach: | They propose a benchmark for evaluating mathematical and physical reasoning through text-only and text-image formats with human-annotated difficulty levels and detailed explanations. |
| Outcome: | The proposed model achieves only 63.77% accuracy and struggles with visual reasoning tasks. |
Selective “Selective Prediction”: Reducing Unnecessary Abstention in Vision-Language Reasoning (2024.findings-acl)
Copied to clipboard
Tejas Srinivasan, Jack Hessel, Tanmay Gupta, Bill Yuchen Lin, Yejin Choi, Jesse Thomason, Khyathi Chandu
| Challenge: | ReCoVERR reduces the over-abstention of a vision-language system with low tolerance for inaccurate predictions without increasing the error rate of the system’s predictions. |
| Approach: | They propose an inference-time algorithm to reduce the over-abstention of a selective vision-language system without increasing the error rate of the system’s predictions. |
| Outcome: | ReCoVERR reduces the over-abstention of a vision-language system without increasing the error rate of the system’s predictions. |
Light Up the Shadows: Enhance Long-Tailed Entity Grounding with Concept-Guided Vision-Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Multi-Modal Knowledge Graphs (MMKGs) are knowledge graphs that integrate and align information from diverse modalities (e.g., text and images). |
| Approach: | They propose a framework that integrates image-text pairs of long-tailed entities and a concept guidance module that offers explainability and enables human verification. |
| Outcome: | The proposed framework improves the accuracy of recognizing long-tailed image-text pairs compared to baselines and also offers flexibility and explainability. |
Boosting Multi-modal Keyphrase Prediction with Dynamic Chain-of-Thought in Vision-Language Models (2025.emnlp-main)
Copied to clipboard
Qihang Ma, Shengyu Li, Jie Tang, Dingkang Yang, null Chenshaodong, Yingyi Zhang, Chao Feng, Ran Jiao
| Challenge: | Multi-modal keyphrase prediction (MMKP) aims to produce concise, informative phrases that capture the essence of cross-modal inputs. |
| Approach: | They propose to use vision-language models to generate conclusive phrases using multiple modalities of input information. |
| Outcome: | The proposed methods outperform existing methods on absence and unseen scenarios and overestimate model capability due to overlap in training tests. |
Interpretable Composition Attribution Enhancement for Visio-linguistic Compositional Understanding (2024.emnlp-main)
Copied to clipboard
| Challenge: | Despite promising progress, vision-language models still exhibit significant challenges in understanding visio-linguistic concepts beyond object terms. |
| Approach: | They propose a framework that encourages the model to pay greater attention to composition words denoting relationships and attributes within the text. |
| Outcome: | The proposed framework improves the ability to discern intricate details and construct more sophisticated interpretations of combined visual and linguistic elements. |
ViStruct: Visual Structural Knowledge Extraction via Curriculum Guided Code-Vision Representation (2023.emnlp-main)
Copied to clipboard
| Challenge: | State-of-the-art vision-language models have limited performance in structural knowledge extraction, such as relations between objects. |
| Approach: | They propose to leverage the inherent structure of programming language to depict visual structural information in a well-organized structured format. |
| Outcome: | The proposed framework improves visual structural knowledge extraction on visual structure prediction tasks. |
LLMs Can Compensate for Deficiencies in Visual Representations (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a strong language backbone in vision-language models compensates for weak visual features by contextualizing or enriching them. |
| Approach: | They investigate whether strong language backbone compensates for weak visual features . they use CLIP-based vision encoders to perform controlled self-attention ablations . |
| Outcome: | The proposed model compensates for weak visual features by contextualizing or enriching them. |
Targeted Exploration via Unified Entropy Control for Reinforcement Learning (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for group relative policy optimization suffer from entropy collapse . Existing exploration methods introduce additional bias or variance during exploration, making it difficult to maintain stability. |
| Approach: | They propose a framework that provides targeted mechanisms for exploration and stabilization. |
| Outcome: | The proposed framework expands search space on difficult prompts while preventing entropy growth uncontrollably. |
Chain-of-Procedure: Hierarchical Visual-Language Reasoning for Procedural QA (2026.findings-acl)
Copied to clipboard
Guanhua Chen, Yutong Yao, Shenghe Sun, Ci-jun Gao, Shudong Liu, Lidia S. Chao, Feng Wan, Derek F. Wong
| Challenge: | Recent advances in vision-language models (VLMs) have achieved impressive results on standard image-text tasks, yet their capability in visual procedure question answering (VP-QA) remains largely unexplored. |
| Approach: | They propose a multimodal benchmark specifically designed for visual procedural reasoning that synergizes cross-modal procedure retrieval, context-aware step decomposition, and the next step prediction. |
| Outcome: | The proposed framework significantly outperforms baselines on visual procedure question answering (VP-QA) Experiments on six VLMs show that it performs better than baselines. |
Debating for Better Reasoning in Vision-Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) gain expertise across diverse domains and modalities, a new study shows . scalable oversight becomes challenging when their capabilities surpass human evaluators. |
| Approach: | a new study extends the debate paradigm to a multimodal setting . it explores the potential for blind models to supervise and enhance the performance of sighted ones. |
| Outcome: | The proposed framework outperforms individual LLMs on multimodal tasks . it allows blind models to supervise and enhance the performance of sighted models . |
Scaling Text-Rich Image Understanding via Code-Guided Synthetic Multimodal Data Generation (2025.acl-long)
Copied to clipboard
Yue Yang, Ajay Patel, Matt Deitke, Tanmay Gupta, Luca Weihs, Andrew Head, Mark Yatskar, Chris Callison-Burch, Ranjay Krishna, Aniruddha Kembhavi, Christopher Clark
| Challenge: | Vision-language models struggle to understand text-rich images due to the scarcity of diverse text-only large language data. |
| Approach: | They propose a framework that leverages the coding capabilities of text-only large language models to create synthetic text-rich multimodal data. |
| Outcome: | The proposed framework can generate high-quality instruction-tuning data using Python, HTML, LaTeX and other languages. |
V-SEAM: Visual Semantic Editing and Attention Modulating for Causal Interpretability of Vision-Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing work on causal interpretability focuses on large language models (LLMs) but internal mechanisms of vision-language models remain underexplored, authors say . |
| Approach: | They introduce a framework that combines visual and semantic manipulations for causal interpretation of vision-language models. |
| Outcome: | The proposed framework shows improved performance for LLAVA and InstructBLIP on three diverse benchmarks. |
When are Lemons Purple? The Concept Association Bias of Vision-Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large-scale vision-language models such as CLIP have shown impressive performance on zero-shot image classification and image-to-text retrieval tasks. |
| Approach: | They propose to use "question text" as input for the text encoder of CLIP to make the prediction harder than it should be. |
| Outcome: | The proposed model treats input as a bag of concepts and attempts to fill in the other missing concept crossmodally, leading to an unexpected zero-shot prediction. |
Retrieval-augmented GUI Agents with Generative Guidelines (2025.emnlp-main)
Copied to clipboard
| Challenge: | GUI agents powered by vision-language models struggle with real-world tasks due to their complex nature and limited training data. |
| Approach: | They propose a lightweight vision-language model that leverages web tutorials at inferencetime to synthesize GUI agents. |
| Outcome: | The proposed agent outperforms baseline GUI agents and surpasses other inference baselines by 2.6% to 13.3% across two model sizes. |
Vision-Flan: Scaling Human-Labeled Tasks in Visual Instruction Tuning (2024.findings-acl)
Copied to clipboard
Zhiyang Xu, Chao Feng, Rulin Shao, Trevor Ashby, Ying Shen, Di Jin, Yu Cheng, Qifan Wang, Lifu Huang
| Challenge: | Recent vision-language models (VLMs) have shown impressive capabilities as general visual assistants, but there are two challenges to their performance: (1) lacking task diversity in pretraining and visual instruction tuning; (2) annotation error and bias in GPT-4 synthesized instruction tuning data. |
| Approach: | They propose a two-stage instruction tuning framework that fine tunes VLMs firstly and further tuned on GPT-4 synthesized data. |
| Outcome: | The proposed framework outperforms the traditional single-stage visual instruction tuning framework and achieves state-of-the-art performance across a wide range of multi-modal evaluation benchmarks. |
Does Visual Grounding Enhance the Understanding of Embodied Knowledge in Large Language Models? (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Despite significant progress in multimodal language models, it remains unclear whether visual grounding enhances their understanding of embodied knowledge compared to text-only models. |
| Approach: | They propose to assess vision-language models’ perceptual abilities across different sensory modalities through vector comparison and question-answering tasks with over 1,700 questions. |
| Outcome: | The proposed benchmark assesses the models’ perceptual abilities across different sensory modalities through vector comparison and question-answering tasks with over 1,700 questions. |
On Efficient Language and Vision Assistants for Visually-Situated Natural Language Understanding: What Matters in Reading and Reasoning (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in language and vision assistants have showcased impressive capabilities but suffer from a lack of transparency, limiting broader research and reproducibility. |
| Approach: | They propose to redefine the design of vision-language models by identifying key components and creating efficient models with constrained inference costs. |
| Outcome: | The proposed models achieve significant improvements in inference throughput while maintaining high performance. |
TransferCVLM: Transferring Cross-Modal Knowledge for Vision-Language Modeling (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent large vision-language multimodal models pre-trained with huge amount of image-text pairs show remarkable performances in downstream tasks. |
| Approach: | They propose a method of efficient knowledge transfer that integrates pre-trained uni-modal models into a combined vision-language model without pre-training . they propose to fine-tune the model and transfer multimodal knowledge from a teacher vision-linguistic model to the CVLM for each task application. |
| Outcome: | The proposed method outperforms existing vision-language models in downstream tasks. |
EgoNormia: Benchmarking Physical-Social Norm Understanding (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing VLMs lack robust grounded norm understanding, a new study finds . current VLM models lack robust grounding, despite a high score for safety and privacy . |
| Approach: | They propose a pipeline to generate grounded MCQs from ego-centric videos of human interactions. |
| Outcome: | The proposed pipeline can generate grounded MCQs from egocentric video . it shows that current VLMs lack robust grounded norm understanding . |
From Descriptive Richness to Bias: Unveiling the Dark Side of Generative Image Caption Enrichment (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have enhanced the capacity of vision-language models to caption visual text. |
| Approach: | They compare standard-format captions and recent GCE processes from the perspectives of gender bias and hallucination. |
| Outcome: | The proposed methods amplify gender bias by 30.9% and increase hallucination by 59.5%. |
Tales of Morality: Comparing Human- and LLM-Generated Moral Stories from Visual Cues (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a recent study has found that stories are central to how humans communicate moral values . |
| Approach: | They compare human- and LLM-generated moral narratives based on images annotated by humans for moral content . authors propose a framework for evaluating moral storytelling in vision-language models . |
| Outcome: | The proposed model compared human- and LLM-generated narratives on images . human stories reflect a balanced distribution of moral foundations and coherent narrative arcs, but LLMs emphasize Care foundation and lack emotional resolution. |
See2Refine: Vision-Language Feedback Improves LLM-Based eHMI Action Designers (2026.acl-long)
Copied to clipboard
| Challenge: | External Human-Machine Interfaces (eHMIs) are emerging as promising solutions to address this communication gap. |
| Approach: | They propose a framework that uses vision-language models (VLMs) for perceptual evaluation as automated visual feedback to improve an LLM-based eHMI action designer. |
| Outcome: | The proposed framework outperforms prompt-only LLM designers and manually specified baselines in three eHMI modalities and multiple LLM model sizes. |
Attack as Defense: Safeguarding Large Vision-Language Models from Jailbreaking by Adversarial Attacks (2025.findings-emnlp)
Copied to clipboard
| Challenge: | adversarial vulnerabilities in vision-language systems pose a challenge to reliability of large systems . typographic manipulations and adversarial perturbations can bypass language model defenses . |
| Approach: | They propose a method that embeds perturbations in vision to disrupt attacks . they use cross-modal interactions to enhance adversarial robustness through perturbations . |
| Outcome: | The proposed approach reduces attack success rates for typographic attacks and adversarial perturbations by integrating visual defenses into the model. |
Puzzled by Puzzles: When Vision-Language Models Can’t Take a Hint (2025.emnlp-main)
Copied to clipboard
| Challenge: | rebus puzzles encode language through imagery, spatial arrangement, and symbolic substitution. |
| Approach: | They construct a benchmark of rebus puzzles in english language to test their ability to interpret and solve them. |
| Outcome: | The proposed model performs well on a set of english-language rebus puzzles. |
Follow the Flow: Fine-grained Flowchart Attribution with Neurosymbolic Agents (2025.emnlp-main)
Copied to clipboard
Manan Suri, Puneet Mathur, Nedim Lipka, Franck Dernoncourt, Ryan A. Rossi, Vivek Gupta, Dinesh Manocha
| Challenge: | Flowcharts are a critical tool for visualizing decision-making processes, but their non-linear structure and complex visual-textual relationships make it difficult to interpret them using LLMs. |
| Approach: | They propose a task of Fine-grained Flowchart Attribution to trace components grounding a flowchart referring LLM response. |
| Outcome: | The proposed agent mitigates visual hallucinations in LLM answers over baselines by 10–14% on a FlowExplainBench dataset. |
Mind the Gesture: Evaluating AI Sensitivity to Culturally Offensive Non-Verbal Gestures (2025.acl-long)
Copied to clipboard
| Challenge: | a dataset of 288 gesture-country pairs is used to evaluate AI systems' cultural awareness of offensive gestures and nonverbal signs. |
| Approach: | They use a dataset of 288 gesture-country pairs annotated for offensiveness, cultural significance, and contextual factors across 25 gestures and 85 countries. |
| Outcome: | The proposed dataset analyzes 288 gesture-country pairs across 25 gestures and 85 countries. |
Socratic-MCTS: Test-Time Visual Reasoning by Asking the Right Questions (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent research in vision-language models has centered around the possibility of equipping them with implicit long-form chain-of-thought reasoning via distillation and reinforcement learning. |
| Approach: | They propose a Monte Carlo Tree Search-inspired algorithm that injects subquestion–subanswer pairs into the model’s output stream to elicit hidden knowledge and induce long reasoning traces. |
| Outcome: | The proposed method yields a 2% improvement on MMMU-PRO, including a significant 9% gain in Liberal Arts. |
Hanfu-Bench: A Multimodal Benchmark on Cross-Temporal Cultural Understanding and Transcreation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on cultural understanding with vision-language models primarily emphasize geographic diversity, often overlooking the critical temporal dimensions. |
| Approach: | They propose a multimodal vision-language model that examines temporal features and cultural image transcreation. |
| Outcome: | The novel model performs better than non-experts on visual cutural understanding but falls short to human experts on cultural image transcreation task. |
WikiMixQA: A Multimodal Benchmark for Question Answering over Tables and Charts (2025.findings-acl)
Copied to clipboard
Negar Foroutan, Angelika Romanou, Matin Ansaripour, Julian Martin Eisenschlos, Karl Aberer, Rémi Lebret
| Challenge: | Documents are fundamental to preserving and disseminating information, often incorporating complex layouts, tables, and charts that pose significant challenges for automatic document understanding (DU). |
| Approach: | They propose a benchmark for evaluating cross-modal reasoning over tables and charts extracted from 4,000 Wikipedia pages . they evaluate 12 vision-language models that achieve 70% accuracy when provided with direct context . |
| Outcome: | The proposed benchmark evaluates models with high accuracy over tables and charts extracted from 4,000 Wikipedia pages . proprietary models achieve 70% accuracy when provided with direct context, but open-source models perform worse when retrieval from long documents is required. |
Walk in Others’ Shoes with a Single Glance: Human-Centric Visual Grounding with Top-View Perspective Transformation (2025.acl-long)
Copied to clipboard
| Challenge: | Existing VLMs are insensitive to information differences induced by slight perspective changes. |
| Approach: | They propose a visual perspective-taking task that requires robots to interpret human-centric instructions and identify corresponding objects from robot perspectives. |
| Outcome: | The proposed method improves performance by up to 18% and generalizes effectively to robotic and dynamic scenarios. |
BannerBench: Benchmarking Vision Language Models for Multi-Ad Selection with Human Preferences (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Web banner advertisements are often selected manually because of human preferences . a new benchmark evaluates the degree of alignment with human preferences in two tasks . |
| Approach: | a benchmark was developed to evaluate the human preference-driven banner selection process using vision-language models. |
| Outcome: | The proposed benchmark assesses the degree of alignment with human preferences in two tasks using vision-language models. |
Jigsaw-Puzzles: From Seeing to Understanding to Reasoning in Vision-Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing vision-language models lack spatial reasoning capability, despite their ability to comprehend spatial arrangements and model structural relations. |
| Approach: | They propose a benchmark to evaluate vision-language models' spatial perception, structural understanding, and reasoning capabilities by minimizing reliance on domain-specific knowledge. |
| Outcome: | The proposed benchmark is based on 1,100 carefully curated real-world images with high spatial complexity. |
Don’t Act Blindly: Robust GUI Automation via Action-Effect Verification and Self-Correction (2026.acl-long)
Copied to clipboard
Yuzhe Zhang, Xianwei Xue, Xingyong Wu, Mengke Chen, Chen Liu, Xinran He, Run Shao, Feiran Liu, Huanmin Xu, Qiutong Pan, Haiwei Wang
| Challenge: | Existing GUI agents assume deterministic environment responses, generating actions without verifying whether previous operations succeeded. |
| Approach: | They propose a GUI agent that explicitly models action outcomes and recovery under noisy environments. |
| Outcome: | The proposed agent reduces failure loops and improves recovery success in noisy environments while maintaining competitive standard task performance. |
FOCUS: Evaluating Pre-trained Vision-Language Models on Underspecification Reasoning (2025.acl-long)
Copied to clipboard
| Challenge: | a new dataset evaluates whether vision-language models have underspecification reasoning abilities . underspecifications are often left incomplete or vague, and are often ignored for mutual understanding . |
| Approach: | They propose a probing dataset to evaluate whether VLMs have underspecification reasoning . they find that pre-trained vision-language models lack this ability . |
| Outcome: | The proposed probing dataset shows that pre-trained vision-language models lack underspecification reasoning abilities. |
Sightation Counts: Leveraging Sighted User Feedback in Building a BLV-aligned Dataset of Diagram Descriptions (2025.acl-long)
Copied to clipboard
| Challenge: | Existing studies show that direct generation of diagram descriptions is costly and biased against blind and low-vision (BLV) users. |
| Approach: | They ask sighted individuals to assess diagram descriptions generated by vision-language models . they use latent supervision to guide the models with latent inference . |
| Outcome: | The results show that visual descriptions generated by vision-language models are effective and useful to educators who are themselves BLV and teach visually impaired learners. |
CheXalign: Preference fine-tuning in chest X-ray interpretation models without human feedback (2025.acl-long)
Copied to clipboard
Dennis Hein, Zhihong Chen, Sophie Ostmeier, Justin Xu, Maya Varma, Eduardo Pontes Reis, Arne Edward Michalson Md, Christian Bluethgen, Hyun Joo Shin, Curtis Langlotz, Akshay S Chaudhari
| Challenge: | Radiologists are a crucial role in translating medical images into actionable reports . however, the field faces staffing shortages and increasing workloads . |
| Approach: | They propose an automated pipeline for preference feedback focusing on chest X-ray radiology report generation (RRG) method leverages publicly available datasets containing pairs of images and radiologist-written reference reports with reference-based metrics, or Judges. |
| Outcome: | The proposed pipeline achieves state-of-the-art CheXbert scores on the MIMIC-CXR dataset while on average maintaining robust performance across six additional image perception and reasoning tasks. |
EXPERT: An Explainable Image Captioning Evaluation Metric with Structured Explanations (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing studies on explainable evaluation metrics generate explanations without standardized criteria and the overall quality of the generated explanations remains unverified. |
| Approach: | They propose a reference-free evaluation metric that provides structured explanations based on fluency, relevance, and descriptiveness. |
| Outcome: | The proposed evaluation template achieves state-of-the-art on benchmark datasets while providing significantly higher-quality explanations than existing metrics. |
Cross-Modal Masked Compositional Concept Modeling for Enhancing Visio-Linguistic Compositionality (2026.acl-long)
Copied to clipboard
| Challenge: | a contrastive learning approach for vision-language models is needed to capture compositional information. |
| Approach: | They propose a framework that masks compositional concepts in one modality and reconstructs them conditioned on full contextual information from the other . |
| Outcome: | The proposed framework enhances compositionality in visual language models and improves their ability to capture syntactic structure and linguistic information. |
A Parameter-Efficient and Fine-Grained Prompt Learning for Vision-Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Current vision-language models extract semantic information from large-scale cross-modal associations, limiting performance and efficiency. |
| Approach: | They propose a detail-oriented prompt learning method to implement fine-grained multi-modal semantic alignment with merely 0.25M trainable parameters. |
| Outcome: | The proposed method implements fine-grained multi-modal semantic alignment with merely 0.25M trainable parameters. |
Are Vision-Language Models Safe in the Wild? A Meme-Based Benchmark Study (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing safety evaluations rely on artificial images to evaluate vision-language models . a recent study found that memes are more effective at bypassing safety measures than synthetic or typographic images. |
| Approach: | They propose a benchmark pairing meme images with harmful and benign instructions . they assess multiple VLMs across single and multi-turn interactions . |
| Outcome: | The proposed benchmark pairs real meme images with harmful and benign instructions. |
ONEBench to Test Them All: Sample-Level Benchmarking Over Open-Ended Capabilities (2025.acl-long)
Copied to clipboard
| Challenge: | ONEBench enables custom benchmarks for specific capabilities while reusing and aggregating samples. |
| Approach: | They propose a new paradigm that consolidates individual evaluation datasets into a unified, ever-expanding sample pool. |
| Outcome: | The proposed model evaluation framework is based on dynamic, sample-level evaluation. |
CausalVLBench: Benchmarking Visual Causal Reasoning in Large Vision-Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large vision-language models have shown impressive ability in various language tasks, especially with their emergent in-context learning capability. |
| Approach: | They propose a causal reasoning benchmark for multi-modal in-context learning from large vision-language models that incorporates visual inputs. |
| Outcome: | The proposed model outperforms existing models on three visual causal reasoning tasks and demonstrates their strengths and weaknesses. |
PluRule: A Benchmark for Moderating Pluralistic Communities on Social Media (2026.acl-long)
Copied to clipboard
| Challenge: | Social media are shifting towards community-governed platforms where groups define their own norms. |
| Approach: | They propose a multimodal, multilingual benchmark for detecting 13,371 rule violations across 1,989 Reddit communities . they show that bigger models and increased context provide marginal gains, and universal rules like civility and self-promotion are easier to detect. |
| Outcome: | The proposed model can detect 13,371 rule violations across 1,989 Reddit communities across 2,885 rules in 9 languages. |
What Do Vision–Language Models Encode for Personalized Image Aesthetics Assessment? (2026.findings-acl)
Copied to clipboard
| Challenge: | Personalized image aesthetics assessment (PIAA) is an important research problem with practical applications. |
| Approach: | They propose a vision-language model that encodes multi-level aesthetic attributes . they analyze visual representations of VLMs to examine their internal representations . |
| Outcome: | The proposed framework can be used to personalize images without fine-tuning . it can be implemented in a variety of image domains and architectures. |
Image Difference Captioning via Adversarial Preference Optimization (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing supervised approaches to image difference captioning overfit to dataset-specific language patterns and fail to capture accurate preferences. |
| Approach: | They propose an adversarial direct preference optimization framework that aligns captioning policy with pairwise difference preferences via Direct Preference Optimization. |
| Outcome: | The proposed approach outperforms baselines on benchmark IDC datasets in generating fine-grained and accurate difference descriptions. |
Thinking Like a Botanist: Challenging Multimodal Language Models with Intent Driven Chain-of-Inquiry (2026.findings-acl)
Copied to clipboard
Syed Nazmus Sakib, Nafiul Haque, Shahrear Bin Amin, Hasan Muhammad Abdullah, Md Mehedi Hasan, Mohammad Zabed Hossain, Shifat E. Arman
| Challenge: | Visual question-based reasoning is a key component of vision-language models. |
| Approach: | They propose a framework for visual question-answering that integrates visual intent with visual severity to improve diagnostic accuracy. |
| Outcome: | The proposed framework improves diagnostic correctness, reduces hallucination, and increases reasoning efficiency. |
Fico: Evaluating Vision-Language Models under Visual Fidelity and Compression at Scale (2026.findings-acl)
Copied to clipboard
| Challenge: | Visual text compression is emerging paradigm for rendering text as images for processing by vision-language models. |
| Approach: | They propose a benchmark to assess VLM robustness under dense visual inputs. |
| Outcome: | Evaluating 13 general-purpose VLMs and 3 OCR-specialized models reveals performance drops sharply under increased density or reduced resolution; cross-task transfer between OCR, NIAH, and VQA is limited; and VQ is comparatively robust because low-level details are lost before high-level semantics. |
CHURRO: Making History Readable with an Open-Weight Large Vision-Language Model for High-Accuracy, Low-Cost Historical Text Recognition (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing vision-language models are not equipped to read diverse languages and scripts found in historical materials. |
| Approach: | They propose to train an open-weight vision-language model for historical text recognition on CHURRO-DS, the largest historical text-recognition dataset to date. |
| Outcome: | The proposed model outperforms existing vision-language models on CHURRO-DS, the largest historical text recognition dataset to date. |
From Heads to Neurons: Causal Attribution and Steering in Multi-Task Vision–Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing models focus on single tasks, limiting comparability of neuron importance . ranking strategies overlook how task-dependent information pathways shape write-in effects of feed-forward network (FFN) neurons. |
| Approach: | They propose a gradient-free framework for task-aware neuron attribution and steering in multi-task vision-language models. |
| Outcome: | The proposed framework outperforms existing methods in identifying task-critical neurons and improves model performance after steering. |
On-policy Reinforcement Fine-tuning with Offline reward for Multi-step Embodied Planning (2026.acl-long)
Copied to clipboard
| Challenge: | Embodied planning requires agents to make coherent multi-step decisions based on dynamic visual observations and verbal goals. |
| Approach: | They propose an On-policy Reinforcement fine-tuning framework with offline rewards for Embodied Task Planning that preserves generalization benefits of RFT while addressing costly interaction and sparse rewards. |
| Outcome: | The proposed framework outperforms closed-source and online-RL methods on EmbodiedBench, a recent benchmark for interactive embodied tasks. |
DANCE: Diversity-attended Dynamic Caching with Asymmetric Quantization for Test-time Adaptation of Vision-Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to test-time adaptation of vision-language models measure prediction entropy but these samples tend to approach prototypes with limited coverage of data distributions. |
| Approach: | They propose a new approach for test-time adaptation of vision-language models . they construct a dynamic cache to store diversity-aware test samples . |
| Outcome: | The proposed approach is more efficient than current methods on augmented visual models. |
Lightweight and Faithful Visual Condition Checking in Behavior Trees via Expert-Regularized Reinforcement Learning (2026.acl-long)
Copied to clipboard
| Challenge: | Existing behavior trees are not suitable for high-dimensional perceptual inputs such as images or language. |
| Approach: | They propose a framework that leverages expert-regularized reinforcement learning to preserve semantic faithfulness while employing a factorized policy that aggregates sequential condition-node decisions into a single decision unit. |
| Outcome: | The proposed framework outperforms imitation learning and reinforcement learning but risks misalignment of condition nodes with intended semantics and poor credit assignment. |
Through the Magnifying Glass: Adaptive Perception Magnification for Hallucination-Free VLM Decoding (2026.acl-long)
Copied to clipboard
| Challenge: | Existing vision-language models suffer from visual hallucination, where the generated responses contain inaccuracies that are not grounded in the visual input. |
| Approach: | They propose a visual decoding method that iteratively isolates relevant visual tokens based on attention and magnifies the corresponding regions. |
| Outcome: | The proposed method reduces language biases and amplifies weights of visual embedding during decoding, while still preserving strong reasoning capabilities. |
Grounded Concreteness: Human-Like Concreteness Sensitivity in Vision–Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | a long tradition in cognitive science treats concreteness as a graded dimension of conceptual representation . concrete words benefit from richer sensory codes and exhibit robust behavioral advantages over abstract words . |
| Approach: | They compare vision-language models with text-only large language models to test their concreteness . they find that VLMs show more human-like sensitivity to concreteness than LLMs . |
| Outcome: | The proposed model-based training improves on the Llama text backbones and Llma Vision counterparts. |
GrAInS: Gradient-based Attribution for Inference-Time Steering of LLMs and VLMs (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning large language models often ignore token-level causal influence and underutilize model logits. |
| Approach: | They propose a novel approach that uses a gradient-based approach to identify influential tokens and construct directional steering vectors based on their contribution to preferred over dispreferred outputs. |
| Outcome: | The proposed approach outperforms fine-tuning and prior steering methods on both LLM and VLM tasks without degrading fluency or general capabilities. |
DREAM-S: Speculative Decoding with Searchable Drafting and Target-Aware Refinement for Multimodal Generation (2026.acl-long)
Copied to clipboard
| Challenge: | Speculative decoding (SD) has proven to be effective for autoregressive generation in large language models (LLMs), however its application to vision-language models (VLMs) remains relatively unexplored. |
| Approach: | They propose a Speculative Decoding framework for vision-language models that integrates a neural architecture search framework and target-aware supernet training to identify optimal interaction strategies. |
| Outcome: | DREAM-S achieves 3.85 speedup compared to baselines on well-established vision-language models. |